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Security  cooperation  professionals  work  towards  the  development  of  a  standardized 
assessment  framework  that  not  only  provides  the  most  complete  data  for 
recommendation  development,  but  also  is  compatible  across  geographic  boundaries 
and  interagency  boundaries.  Across  the  inter-agency  a  common  language  does  not 
exist  when  referring  to  the  planning  and  assessment  process.  In  order  to  maximize  the 
efficiency  of  US  Government  programs  in  security  cooperation  around  the  globe,  there 
must  be  some  efforts  to  not  only  improve  assessment,  but  also  to  build  some 
commonality  among  assessment  frameworks  and  progress  measures.  This  paper  aims 
to  highlight  some  considerations  when  planning  and  implementing  security  cooperation 
and  should  provide  the  reader  with  some  critical-thinking  tools  to  continue  discussion 
with  interagency  partners  involved  in  security  cooperation.  The  paper  discusses 
assessment  based  on  current  policy,  considers  strengths  and  weaknesses  of  three 
assessment  or  research  methodologies,  then  current  practices  and  opportunities.  Finally 
it  provides  some  recommendations  for  further  academic  study  and  discussion. 


Assessing  Security  Cooperation:  Improving  Methods  to  Maximize  Effects 

I  have  urged  the  Department  to  develop  innovative  approaches  to  meeting 
future  security  challenges,  approaches  that  take  better  advantage  of  the 
opportunities  for  partnership  and  help  us  to  more  effectively  advance  a 
common  security  vision  for  the  future.  To  that  end,  I’ve  directed  all  of  the 
geographic  Combatant  Commanders  to  think  and  plan  strategically  when 
it  comes  to  security  cooperation... 

— The  Honorable  Leon  Panetta,  Secretary  of  Defense1 
Assessment  is  a  key  aspect  of  military  planning.  This  is  especially  true  for 
ongoing  operations,  as  the  assessment  mechanics  typically  indicate  where  changes 
and  adjustments  need  to  occur.  The  United  States  Military  is  perhaps  one  of  the  best 
planning  organizations  in  the  world,  and  throughout  combat  operations,  military 
planners  show  an  impressive  tenacity  for  being  able  to  adapt  to  changes  in  the 
operational  environment.  That  tenacity  comes  from  the  timely  and  accurate 
assessments  made  on  a  regular  basis  underpinned  by  key  measurements  and  critical 
thinking.  When  it  comes  to  security  cooperation,  the  level  of  quality  in  ongoing 
assessment  may  not  be  as  rigorous.  Perhaps  it  is  because  there  are  not  as  many 
established  “metrics”  to  provide  indications  of  success  (or  failure).  Perhaps  it  is  due  to 
the  fact  that  security  cooperation  can  in  many  ways  be  very  “relational.”  Monitoring 
progress  or  success  of  many  actions  and  programs  in  security  cooperation  can  be 
difficult  because  measures  such  as  “improved  trust,”  “capacity,”  and  “partnership”  do  not 
fit  well  into  spreadsheets.  Security  cooperation  professionals  must  work  towards  the 
development  of  a  standardized  assessment  framework  that  not  only  provides  the  most 
complete  data  for  recommendation  development,  but  also  is  compatible  across 
geographic  boundaries  as  well  as  across  inter  agency  boundaries. 


With  the  future  of  fiscal  challenges  and  increased  importance  of  Building 
Partnership  Capacity  (BPC)  and  security  cooperation,  accurately  assessing  progress 
would  provide  input  to  decision-makers  that  may  better  ensure  efficiency  and 
effectiveness.  Additionally,  across  the  interagency,  a  common  language  does  not  exist 
when  referring  to  the  planning  and  assessment  process.  In  order  to  maximize  the 
efficiency  of  US  Government  security  cooperation  programs  in  countries  around  the 
globe,  there  must  be  some  unified  efforts  to  not  only  improve  assessment,  but  also  to 
build  some  commonality  among  agency  assessment  frameworks  and  progress 
measures.  By  developing  some  framework  commonality,  planners  would  ensure  that 
military  strategies  are  in  line  with  diplomacy  and  development  strategies  -  all  nested 
under  the  national  interest.  If  each  country  team  in  a  geographic  region  uses  different 
methods  and  processes,  then  combining  data  at  the  regional  level  could  lack 
consistency  and  meaning,  or  worse  provide  erroneous  data  resulting  in  poor  decisions 
and  choices  when  updating  our  security  cooperation  portfolio. 

This  paper  aims  to  highlight  some  considerations  when  planning  and 
implementing  security  cooperation.  The  paper  begins  with  a  discussion  on  the  rationale 
for  such  assessment  considerations  based  on  current  policy  documents.  It  then 
considers  strengths  and  weaknesses  of  three  assessment  or  research  methodologies 
from  academia:  quantitative,  qualitative,  and  mixed-method.  From  there  the  discussion 
moves  to  some  current  processes  in  place  as  well  as  some  opportunities  for  developing 
assessments  to  inform  decision-makers.  The  paper  then  combines  these  to  provide 
some  recommendations  and  a  road  ahead  for  further  study  and  consideration.  Ideally, 
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this  should  provide  the  reader  with  some  critical-thinking  tools  to  continue  discussion 
with  interagency  partners  involved  in  security  cooperation. 

Why  Improve  Methods  for  Assessment? 

In  the  security  cooperation  planning  process,  much  thought  is  given  to  ensuring 
that  a  given  country  campaign  plan  (CCP)  nests  under  the  Geographic  Combatant 
Command  (GCC)  Theater  Campaign  Plan  (TCP).  There  is  also  consideration  that  it 
complies  with  the  Embassy’s  Mission  Strategic  Resource  Plan  (MSRP).2  There  is  much 
less  documentation  that  discusses  the  requirements  and  methodology  for  assessing 
these  plans  to  complete  the  assess-plan-execute  loop.  In  periods  where  budgets  may 
be  limited  and  increased  scrutiny  given  to  how  all  funding  is  allocated,  it  will  become 
increasingly  important  for  security  cooperation  professionals  to  have  effective 
assessment  measures  in  place  to  quickly  assess  the  most  effective  programs  exist,  and 
where  risk  may  be  greatest. 

The  Guidance  for  the  Employment  of  the  Force  (GEF)  and  Joint  Strategic 
Capabilities  Plan  (JSCP)  documents  direct  that  Combatant  Commanders  (CCMD) 
establish  ongoing  “living”  campaign  plans  that  “integrate  security  cooperation,  Phase  0, 
and  other  steady-state  activities.”3  Equally  important  in  this  guidance  is  that  these 
campaign  plans  are  to  provide  a  method  to  conduct  “comprehensive  assessments”  of 
how  the  plans  (in  execution)  are  contributing  towards  the  accomplishment  of  the  theater 
specific  end  states  outlined  in  the  GEF.  As  the  Commander  is  required  to  report  on  this 
assessment,  and  incorporate  it  into  his  annual  Comprehensive  Joint  Assessment 
(CJA)4,  it  follows  that  there  must  be  some  mechanism  in  place  that  provides  this 
assessment,  and  some  indicators  built  into  the  plan  that  provide  progress  information. 
The  security  cooperation  team  must  consider  this  in  the  initial  planning  phase,  must 
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monitor  during  execution,  and  must  be  prepared  to  recommend  changes  based  on 
legitimate  assessment. 

If  the  TCP  were  a  unilateral  Department  of  Defense  (DoD)  document,  then 

perhaps  this  would  be  a  simpler  task,  however  this  plan  must  be  closely  coordinated 

with  the  Department  of  State  (DoS)  and  the  United  States  Agency  for  International 

Development  (USAID).  The  challenge  in  coordinating  different  planning  schedules, 

timelines,  and  languages  is  significant.  From  the  draft  3D  Planning  guide: 

Within  the  loop  of  assessing,  planning,  and  implementing  (variations  on 
this  exist  within  each  of  the  organizations),  the  lack  of  common 
assessment  frameworks  presents  a  problem:  as  each  agency  uses  a 
different  set  of  tools  and  lenses  to  assess  problems,  there  is  a  potential  to 
plan  based  upon  differing  assumptions.  This  can  result  in  unsynchronized 
activity  and  divergent  goals  at  the  country  level.  This  lack  of 
synchronization  can  seriously  dilute  the  overall  effectiveness  of  the  U.S. 
Government  effort.5 

The  planning  guide  goes  on  further  to  describe  how  the  methodology  for 
assessment,  monitoring  and  evaluation  differs  between  these  key  agencies.  While  no 
one  disagrees  that  there  is  need  to  analyze  throughout  a  plan’s  development  and 
implementation,  it  is  essential  that  all  involved  understand  these  organizational 
differences.  As  Carl  Builder  notes,  “Institutional  and  personal  interest  are  not  intrinsically 
bad;  but  they  may  be  made  so  if  they  are  always  cloaked  in  altruism  and  not 
acknowledged  as  legitimate  interests.”6 

If  the  planning  process  results  in  mutually  agreed  upon  metrics,  then  there  is 
potential  that  collected  assessment  data  is  useful  for  multiple  organizations.  As  a  sort  of 
“force  multiplier,”  the  assessment  requirement  becomes  easier  if  the  military  security 
cooperation  professional  can  utilize  data  that  is  collected  by  someone  else  -  both  the 
plans  and  the  assessment  benefit  from  interagency  synchronization.  The  Office  of  the 
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Secretary  of  Defense  (OSD)  and  Joint  Staff  provide  one  forum  for  coordination,  known 
as  Promote  Cooperation  (PC).7  It  could  be  argued  that  coordination  at  this  high  level 
(Washington,  DC),  while  essential,  is  insufficient  to  truly  bring  these  plans  into 
synchronization.  The  inter-agency  coordination  begins  at  the  country  team  level. 

Continual  assessment,  with  performance  metrics/indicators  incorporated  into  the 
initial  plan  provides  an  additional  benefit:  it  ensures  that  coordinated  actions  across  all 
agencies  are  obtaining  the  desired  results.  While  this  may  currently  occur,  there  are 
instances  where  it  appears  that  results  have  been  less  than  satisfactory.  In  his  most 
recent  book,  Little  America:  The  War  Within  the  War  for  Afghanistan,  Rajiv 
Chandrasekaran  describes  a  facet  of  the  Afghanistan  development  plan  executed 
through  USAID,  the  Afghanistan  Vouchers  for  Increased  Production  in  Agriculture 
(AVIPA).  Modeled  after  a  program  in  northern  Afghanistan,  AVIPA  would  provide  farmer 
subsidies  for  melon  and  vegetable  production  -  an  effort  to  decrease  the  popularity  of 
poppy  farming.  Focused  in  the  Helmand  and  Kandahar  provinces,  USAID  proposed 
spending  $150  in  just  one  year,  AMB  Holbrooke  went  further:  “double  it.”8 

Chandrasekaran  goes  on  to  describe  how  AVIPA  morphed  into  a  cash-for-work 
program,  where  the  main  goal  was  to  get  the  $300M  spent.  The  effect  in  one  district, 
Nawa,  was  a  sudden  influx  of  expendable  cash  in  the  population  ($400  for  every  man 
woman  and  child  over  the  year)9.  Now  residents  were  getting  day  wages  to  clean  out 
irrigation  ditches,  something  they  would  have  collectively  done  without  pay.  This 
achieved  the  goal  of  making  it  more  lucrative  for  people  to  day-labor  than  to  work  for  the 
Taliban,  but  in  the  end  it  did  not  meet  the  goal  of  strengthening  local  government.  With 
massive  projects  required  to  maintain  the  “burn  rate,”  most  cash  went  to  American 
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development  contractors,  who  hired  expensive  Americans  or  foreigners,  who  required 
security.  In  the  end,  security,  management  and  overhead  costs  spent  seventy  percent 
of  the  contract  values  -  only  thirty  cents  of  each  dollar  was  making  it  to  help  Afghans.10 1 
believe  this  case,  while  extreme,  is  an  example  of  why  the  security  cooperation 
professional  must  consider  their  assessment  prior  to  execution,  and  be  ready  to  make 
adjustments  if  stated  goals  are  not  being  met. 

Assessment  Methodologies 

It  is  valuable  to  briefly  look  at  several  methodologies  used  in  analytic  research. 
While  the  focus  of  this  paper  is  not  to  instruct  how  to  develop  a  research  plan  based  on 
any  of  these  methodologies,  it  is  necessary  for  the  security  cooperation  professional  to 
understand  some  of  the  basics  of  each,  as  well  as  give  thought  to  the  associated 
strengths  and  weaknesses.  As  mentioned  above,  when  working  within  the  inter¬ 
agency’s  Diplomacy,  Development,  and  Defense  planning  process  (or  3D)  not  all 
agencies  view  “assessment”  in  the  same  way.  The  3D  planning  guide  offers  the 
following  definitions: 

Assessment  (DoD):  1 .  A  continuous  process  that  measures  the  overall 
effectiveness  of  employing  joint  force  capability  during  military  operations. 

2.  Determination  of  the  progress  toward  accomplishing  a  task,  creating  an 
effect,  or  achieving  an  objective.  (JP  1-02) 

Assessment  (USAID):  The  analysis  and  critical  evaluation  of  pre-existing 
environmental,  political,  sociological,  cultural  or  other  conditions  or 
situations  which  would  have  an  effect  upon  or  influence  the  success  of  a 
program  or  achievement  of  a  Development  Objective.11 

Clearly,  there  is  a  difference  in  perspective  between  the  two  definitions:  one 

looks  to  “measure”  effectiveness  or  accomplishment,  while  the  other  looks  to  “critically 

evaluate”  conditions  that  may  affect  success.  To  be  clear,  there  is  utility  in  each,  and 

potential  exists  for  each  to  strengthen  the  other.  A  basic  understanding  of  some 
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research  and  assessment  methods  from  academic  traditions  may  lead  to  improved 
communication  and  ultimately  more  thorough  plan  development. 

Quantitative  Methodology 

The  quantitative  assessment  methodology  seems  to  be  the  principal  choice  for 
assessment  throughout  the  Department  of  Defense  (DoD).  Perhaps  this  is  due  to  the 
fact  that  there  are  “hard  numbers”  traceable  back  to  a  particular  source.  The  desire  for 
these  metrics  (i.e.,  Measures  of  Effectiveness12,  Performance13,  and  Suitability14) 
requires  something  that  is  countable  -  something  tangible.  Webster’s  dictionary  defines 
quantitative  as,  “of,  relating  to,  or  expressible  in  terms  of  quantity...  involving  the 
measurement  of  quantity  or  amount.”  Research  using  quantitative  assessment  seeks  to 
use  mathematical  or  statistical  modeling  to  better  understand  or  describe  the  data,  and 
often  write  about  it  in  an  impersonal,  third  person  way.15 

One  argument  for  quantitative  methods  is  repeatability.  If  the  data  is  collected 
correctly  and  accurately,  then  any  reviewer  should  be  able  to  come  to  the  same 
conclusion  or  result.  Repeatability  should  also  be  possible  if  the  data  were  collected  by 
multiple  independent  sources.  Due  to  the  ability  to  mathematically  or  statistically  model, 
there  is  little  need  for  interpretation  other  than  trend  analysis,  path,  regression,  etc. 

Data  quality  is  paramount  to  this  process,  and  often  is  the  downfall  of  untrained 
practitioners.  There  is  an  adage  that,  “you  often  can’t  count  what  matters,  and  what  you 
can  count  doesn’t  matter.”  Care  must  be  taken  by  the  planner  that  the  data  collected 
quantitatively  actually  means  something  in  the  end.  Think  back  to  the  AVIP  example 
discussed  earlier  -  the  dollars  spent  was  not  the  right  value  to  collect  for  a  measure  of 
effectiveness  (MOE).  While  it  may  have  value  as  a  measure  of  performance  (MOP) 

which  could  “contribute”  to  an  MOE,  misidentification  led  to  poor  decisions  and  choices. 
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Also  critical  to  data  quality  is  the  need  for  purity  and  completeness.  There  must 
be  no  “guesstimation”  or  “fudge  factor”  that  needs  explanation.  Care  must  be  taken  to 
collect  and  represent  all  data,  not  just  that  which  contributes  to  the  desired  outcome. 
Ethical  failures  in  data  collection  caused  by  political  or  performance  motives  may 
undermine  resulting  recommendations.  The  data  must  stand  on  its  own  -  any  evidence 
of  data  manipulation  will  question  the  validity  and  pedigree  of  the  assessment.  To  this 
end,  many  larger  educational  bodies  establish  rules  and  guidelines  for  data  quality  -  for 
example,  the  Environmental  Protection  Agency  takes  great  care  in  data  quality 
standards  to  ensure  accurate,  repeatable  research. 

Taking  all  these  factors  into  account  in  regard  to  security  cooperation  planning, 
the  quantitative  method  may  provide  some  stumbling  blocks.  While  there  are  some 
items  that  can  be  quantified  (i.e.  number  of  weapon  systems,  number  of  days  of  military 
exercise,  number  of  engagements  with  senior  leaders,  completion=  yes/no),  there  may 
be  difficulty  planning  enough  to  accurately  assess  the  effectiveness  of  a  given  action  or 
plan.  The  Department  of  State  uses  Foreign  Assistance  Indicators16  which  the  Security 
Cooperation  Office  should  be  able  to  access  and  incorporate  in  the  planning  process, 
however,  these  indicators  may  not  provide  data  that  is  meaningful  to  military 
assessments.  It  should  be  noted  that  to  collect  this  data,  the  collection  plan  needs  to  be 
in  place  prior  to  execution,  as  accurate  data  collection  may  be  difficult  to  accomplish 
post  fact.  Reconstructing  data  to  justify  or  disprove  a  specific  effort  risks  bias  in  data 
collection  -  a  “collect  what  you  want”  mentality.  As  mentioned  above,  data  quality  and 
completeness  is  paramount  to  the  validity  of  the  assessment. 
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Still,  quantifiable  data  provides  a  valuable  tool  for  determining  if  dollars  or  effort 
are  wisely  used  in  some  cases.  If  properly  planned  and  collected,  it  may  be  compared 
to  similar  data  in  other  countries  or  regions.  The  security  cooperation  professional, 
however,  is  often  left  with  a  predominance  of  anecdotal,  conversational,  or  interpreted 
data  -most  effectively  utilized  in  a  qualitative  way. 

Qualitative  Methodology 

If  the  security  cooperation  planner  does  not  have  metrics  that  can  be  counted, 
they  are  driven  towards  qualitative  assessment,  where  the  data  collected  is  often 
anecdotal  and  relational.  In  the  academic  world,  the  qualitative  researcher  may  be 
described  as  journalistic,  unscientific,  exploratory,  or  “entirely  personal  and  full  of 
bias.”17  This  is  in  complete  contrast  to  what  was  described  above  as  quantitative 
method.  Traditionally,  in  academia,  this  contrast  creates  conflict  and  both  sides  will 
argue  that  the  other  is  missing  essential  points  of  analysis. 

Lack  of  repeatability  is  a  common  critique  of  the  qualitative  method  in  that  the 
process  is  interpretive  and  endlessly  creative,  relying  on  the  understanding,  experience, 
and  relationships  of  the  researcher.18  In  collecting  and  analyzing  material,  there  are 
several  basically  accepted  practices  and  methods:  interview,  observation,  reading 
material  culture  and  its  records,  visual,  personal  experience,  and  narrative/content 
analysis.19  As  one  can  see  through  these  methods,  there  is  reason  for  the  critique  of 
repeatability,  however  there  is  important  assessment  that  can  be  accomplished. 
Interviews,  news  articles,  memorandums,  and  personal  observations  provide  often 
necessary  context  to  assessment,  particularly  where  “countable”  metrics  cannot  be 
used.  Consider  the  shortcomings  of  the  metrics  applied  in  the  AVIPA  example.  While 
dollars  were  being  spent,  the  qualitative  data  told  another  story. 
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One  common  concern  in  qualitative  methodology  is  the  influence  of  politics  and 
ethics  in  data  collection.  In  the  academic  disciplines,  there  has  even  been  a  push  for 
“certification”  before  a  researcher  is  allowed  to  do  field  research.20  While  this  is  certainly 
not  necessary  for  the  purposes  of  TCP  or  CCP  assessment,  it  does  bring  to  light  the 
potential  for  serious  problems  with  qualitative  data.  How  easy  would  it  be  to  merely 
report  the  observations  that  contribute  to  the  outcome  the  organization  desires,  while 
“overlooking”  those  that  do  not?  If  the  assessment  plan  developed  relies  on  qualitative 
data  -  requiring  some  method  of  personal  collection  -  then  there  is  an  ethical 
requirement  to  collect  as  complete  a  data  field  as  possible. .  .the  good  and  the  bad. 
Performance  requirements  may  tempt  assessors  to  “polish”  a  situation;  however,  those 
politics  may  in  turn  be  indefensible.  In  regards  to  the  political  lens,  it  is  also  important  to 
remember  that  (as  mentioned  earlier)  in  an  interagency  setting  there  will  always  be 
legitimate  interests  held  by  institutions.  It  is  important  to  carefully  consider  qualitative 
data  from  another  source  and  consider  what  prejudices  may  exist. 

Qualitative  methods  may  also  have  limitations  on  data  quality.  As  already 
discussed,  the  process  is  interpretive  and  relies  on  understanding,  experience,  and 
relationships.  It  follows,  then  that  quality  of  data  will  improve  the  longer  the  security 
cooperation  professional  is  in  a  given  position.  Combined  with  the  average  posting 
length  of  3  years,  this  raises  concern.  If  data  quality  fluctuates  and  is  non-repeatable, 
how  accurate  will  the  Combatant  Commander  expect  it  to  be?  If  the  assessment  is 
ultimately  a  hunch  anchored  on  personal  experience,  then  any  recommendations  for 
change  may  not  be  considered  worthwhile  compared  to  others  based  on  performance 
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numbers.  This  perhaps  points  to  why  the  United  States  Government  prefers  to  rely  on 
quantitative  methods. 

In  regards  to  security  cooperation,  I  believe  it  is  clear  that  there  is  utility  for 
qualitative  methods  in  assessment.  The  strength  of  information  gained  from  interview  or 
observation  may  lend  a  critical  context  to  either  success  or  failure  of  a  given  activity. 
Additionally,  this  interpretive  data  may  point  to  a  process  improvement  based  on  local 
cultural  norms  and  history,  or  societal  understanding.  In  the  same  way  that  the 
Constructivist  theory  seeks  “thick  description  as  a  form  of  explanation,”  and  insists  that 
historical  and  social  constructs  matter21,  so  may  qualitative  methods  provide  a  social 
and  historical  context  to  improving  Security  Cooperation.  Still,  it  is  not  worth  shedding 
the  data  which  can  be  counted,  which  brings  us  to  a  third  option  in  methodology. 

The  “Via  Media”  -  Mixed  Methodology 

In  Social  Science  research,  mixed  methodology  is  the  middle-road.  It  is  the 
careful  combination  of  qualitative  and  quantitative  methods  to  enhance  the  available 
data.  The  emergence  of  mixed  methodology  in  Social  Sciences  occurred  in  the 
1980’s.22  In  addition  to  the  value  added  from  qualitative  data  providing  context  to  the 
quantitative,  there  are  other  benefits  to  this  approach.  Validity  may  be  improved  by 
allowing  both  types  of  data  to  corroborate  one  another.  Not  only  may  the  data  be  more 
comprehensive,  but  it  may  increase  credibility  of  findings,  as  well  as  potentially  provide 
explanation  for  results.  Previous  sections  discussed  the  strong  points  of  both  of  these 
methods,  and  mixed  methodology  for  assessment  collection  may  pull  out  the  best  of 
both,  filling  in  critical  knowledge  gaps. 

In  the  process  of  mixing  data,  two  processes  stand  out  as  particularly  applicable 

to  potential  security  cooperation  assessment.  First  is  converging  data,  taking  qualitative 
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data  (numbers,  frequency,  etc.)  and  combining  with  qualitative  data  (anecdote,  news, 
observation)  to  arrive  at  a  result.  An  example  of  this  could  be  an  increase  in  the  number 
of  weapon  system  “x”  (quantitative)  combined  with  an  observed  “improvement”  in 
interoperability  (qualitative).  This  could  lead  to  the  interpretation  that  foreign  military 
sales  (FMS)  case  for  “x”  successfully  builds  capacity  and  interoperability  -  a  result  that 
might  not  be  achieved  without  converging  data. 

The  second  method  is  connecting  data,  where  the  results  or  interpretation  are 
achieved  either  by  using  quantitative  data  and  then  following  up  with  qualitative 
assessment,  or  using  qualitative  data  to  develop  a  quantitative  metric  for  assessment. 
For  one  example,  using  an  International  Military  Education  and  Training  (IMET)  case,  a 
country  is  able  to  send  “x”  personnel  to  a  training  program  (quantitative).  Those 
personnel  participate  in  a  follow-up  interview  (qualitative)  and  it  is  determined  that  even 
though  the  course  was  provided,  the  capacity  still  is  not  there  (based  on  interview  and 
observation).  In  another  example,  through  discussion  it  is  revealed  that  the  country’s 
targeting  effectiveness  has  greatly  improved  (qualitative)  building  to  develop  a 
quantitative  metric  of  “x”  successes.  In  this  instance  an  FMS  case  can  be  linked  and 
attributed  to  the  success. 

Some  of  these  quantitative  or  qualitative  variables  currently  exist,  but  they 
perhaps  are  not  being  leveraged  to  provide  the  best  assessment.  In  example,  the 
Defense  Security  Cooperation  Agency  (DSCA)  has  deployed  the  Regional  Centers 
Person  and  Activity  Management  System  (RCPAMS)  to  provide  support  for  tracking  and 
reporting  purposes  related  to  the  Regional  Centers  it  operates.23  While  this  is  primarily  a 
record  keeping  method  that  integrates  data  between  Security  Cooperation  Offices 
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(SCOs)  and  Regional  Centers24  for  reporting  to  entities  such  as  congress,  the 
quantitative  data  may  be  useful  in  guiding  some  additional  qualitative  follow-up  to 
further  refine  effectiveness  and  application  for  the  security  cooperation  planner.  For 
example,  based  on  qualitative  collection,  are  objectives  better  met  through  “resident 
courses”  or  through  mobile  training-teams  (MTTs)? 

Clearly,  there  is  utility  to  mixed  methodology  assessment.  The  problem  is  in 
combining  into  a  useable  and  meaningful  form.  It  can  be  argued  that  much  of  this  simply 
cannot  be  planned  ahead  of  execution,  and  that  it  results  in  a  reactive  collection  plan. 
However,  if  the  planning  process  evolves  to  include  mechanisms  for  identifying 
indicators  of  both  quantitative  and  qualitative  sources,  then  the  security  cooperation 
professional  may  be  able  to  lay  the  groundwork  for  validating  or  disqualifying  actions 
based  on  both  countable  metrics  and  anecdotal  observation. 

Current  Practices  in  Place 

OSD/JCS  Processes  in  Place 

Currently  the  Joint  Staff  maintains  oversight  of  the  planning  process  directed 
through  the  JSCP  and  conducted  using  the  Adaptive  Planning  and  Execution  (APEX) 
system,  instituted  in  201 1 .  Thorough  guidelines  of  the  planning  timelines  and 
requirements  can  be  found  in  the  Chairman,  Joint  Chiefs  of  Staff  Instruction  (CJCSI) 

31 41 .01 E  as  well  as  Joint  Publication  5-0,  Joint  Operation  Planning.  For  TCP  In¬ 
progress  Reviews  (IPRs),  the  guidance  requires,  “a  focus  on  how  the  CCMD  will 
measure  plan  achievement  of  global/theater  end  states.  In  addition,  key  aspects  of  how 
the  achievement  is  impacted  by  theater  engagement  concept,  security  cooperation, 
country  engagement,  interagency  planning,  and  multinational  planning.”25 
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Aside  from  establishing  timelines  leading  to  plan  approval,  CJCSI  3141 .01 E  also 
discusses  the  use  of  IPR-R,  which  is  for  review  of  plans  in  execution  (TCPs).  At  the 
highest  level,  this  is  the  opportunity  for  the  Commander  to  discuss  the  direction  of  future 
planning  with  the  Secretary  of  Defense  (SECDEF).  Also,  based  on  assessments,  the 
Commander  also  provides  a  RATE  recommendation  (refine,  adapt,  terminate,  and 
execute).26  As  the  TCP  is  already  in  execution  and  will  not  be  terminated,  the  RATE 
recommendation  should  focus  on  refinements  and  adaptations  -  again  based  on 
assessment  findings.27  Through  this,  we  can  see  that  there  is  an  avenue  for  input  of 
assessment  findings  into  the  planning  process.  Still,  to  be  a  valid  recommendation,  that 
assessment  must  have  some  pedigree  linked  back  to  either  quantitative  or  qualitative 
metrics  (or  both). 

One  facet  of  the  current  methodology  described  in  Joint  Operational  Planning  is 
the  cognitive  framework  of  operational  design.  Ultimately,  operational  design  is  a 
“process  of  iterative  understanding  and  problem  framing  that  supports  commanders  and 
staffs  in  their  application  of  operational  art  with  tools  and  a  methodology  to  conceive  of 
and  construct  viable  approaches  to  operations  and  campaigns.”28  Through  operational 
design,  the  commander  (and  their  staff)  should  develop  a  better  understanding  of  the 
operational  environment  across  the  political,  military,  economic,  social,  informational, 
and  infrastructure  (PMESII)  disciplines,  which  additionally  requires  a  better 
understanding  of  how  other  agencies  may  influence  particular  areas.  This  process 
requires  frequent  revisiting  to  ensure  that  an  understanding  of  the  changing 
environment  is  reflected  in  both  problem  definition  and  operational  approach.  It  would 
follow  that  this  understanding  would  not  only  provide  the  critical  inputs  to  the  IPR-R  and 
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the  associated  RATE  recommendation,  but  also  would  improve  the  ability  to 
communicate  in  the  inter-agency  on  more  than  military  specific  goals.  Through 
operational  design,  the  application  of  operational  art  “promotes  unified  action  by  helping 
Joint  Force  Commanders  (JFCs)  and  staffs  understand  how  to  facilitate  the  integration 
of  other  agencies  and  multinational  partners  toward  achieving  strategic  and  operational 
objectives.”29  Interestingly,  this  process  of  defining  the  environment  and  recognition  of 
trends  and  relationships  will  require  incorporation  of  many  qualitative  factors  -  a  cultural 
shift  for  the  military  desire  for  the  quantifiable,  but  useful  for  security  cooperation 
assessment. 

Another  opportunity  for  the  Commander  to  provide  campaign  plan 
recommendations  is  through  the  Joint  Staff  J5  (JS/J5)  run  Comprehensive  Joint 
Assessment  (CJA).  Produced  by  the  Combatant  Command  staff,  it  includes  an 
assessment  of  progress  towards  meeting  GEF  defined  end  states  and  Intermediate 
Military  Objectives  (IMOs).  In  this  sense,  campaign  plan  assessments  can  provide  an 
indication  of  how  security  cooperation  activities  contribute  towards  achieving  theater 
end  states.  Security  cooperation  activities  are  also  discussed  in  their  own  section  and 
recommendations  could  be  included  here  as  well.  In  the  CJA,  for  consistency,  all  inputs 
must  clearly  identify  the  recommendation  and  define  the  responsible  DoD  or  inter 
agency  office  (this  is  a  recommendation,  not  tasking  to  other  agencies).  Then  that 
recommendation  must  be  categorized  within  the  Doctrine,  Organization,  Training, 
Materiel,  Leadership  and  Education,  Personnel,  Facilities,  and  Policy  (DOTMLPF) 
framework,  and  “clearly  identify  key  assessment  findings  that  serve  as  evidence  for  the 
recommendation.”30  This  is  an  opportunity  for  adjustment  recommendations,  but 
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requires  justification  by  assessment  findings  -  gained  through  carefully  developed 

assessment  plans  and  implementation  of  the  operational  design  cognitive  framework. 

Department  of  State  and  the  Integrated  Country  Plan 

Earlier,  this  paper  discussed  some  of  the  observations  of  the  “3D  Planning 

Guide”.  The  3D  Planning  Group  (3DPG)  was  chartered  with  members  of  the  DoS, 

USAID,  and  DoD  -  the  3  pillars  of  Diplomacy,  Development  and  Defense  (3D)  in  order 

to  develop  products  and  processes  to  improve  collaboration  in  planning.31  This  paper 

has  also  shown  that  the  3DPG  recognizes  that  differences  exist  in  defining  exactly  what 

an  assessment  is.  The  DoS  has  teamed  with  USAID  in  development  of  the  Integrated 

Country  Strategy  (ICS)  construct.  Per  the  documentation,  this  ICS,  “sets  Mission  Goals 

and  Mission  Objectives  through  a  coordinated  and  collaborative  planning  effort 

among... US  Government  agencies  operating  overseas.”32 

The  structure  of  this  ICS  starts  with  Mission  Goals,  supported  by  Mission 

Objectives,  each  of  which  contains  an  action  plan.  The  guidance  document  describes 

the  Mission  Objectives  as  “concrete,  realistic  statements  of  a  result  in  the  host  country 

that  the  Mission  can  significantly  influence.”33  Perhaps  most  relevant  to  the  topic  of  this 

paper  is  the  significant  thought  given  to  “measuring  success.”  The  ICS  framework  lays 

out  guidelines  for  measuring  success  using  already  existing  performance  indicators,  for 

example  the  standard  Foreign  Assistance  Indicators  that  the  Mission  already  reports  on 

annually.  In  the  case  that  there  is  no  performance  indicator  already  developed,  the 

Mission  is  required  to  describe  how  progress  will  be  monitored: 

Proactively  thinking  about  ways  to  monitor  progress  during  the  planning 
state  allows  Missions  to  build  ongoing  monitoring  practices  into  their 
contracts  or  routines  that  can  later  be  used  to  satisfy  reporting 
requirements  such  as  the  Foreign  Assistance  Performance  Plan  and 
Report,  Mission  Resource  Request,  or  other  regular  data  calls.34 
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Similar  to  the  DoD  documents  there  is  a  requirement  for  assessment,  but  the  ICS 
guidance  takes  it  one  step  farther  in  requiring  identification  of  monitoring  methods. 
Inclusion  of  this  rigor  into  the  CCP  planning  process  could  provide  the  security 
cooperation  professional  with  some  ready  assessment  data  for  feedback  to  the  TCP 
planning  loop,  or  “other  regular  data  calls.” 

As  part  of  the  country  team,  the  Security  Cooperation  Office  (SCO)  should 
already  be  involved  in  the  development  of  the  ICS,  providing  valuable  input  into  the 
security  section  of  the  plan.  It  follows  that  these  personnel  will  continue  to  become  more 
familiar  with  the  measurement  tools  such  as  the  Foreign  Assistance  Indicators.  If  these 
provide  sufficient  measurement,  and  the  CCP  must  reflect  the  goals  of  the  ICS,  then 
this  methodology  may  become  transparent  once  the  ICS  construct  is  fully  integrated  into 
all  geographic  regions.35  The  ICS  plans  will  be  aligned  with  the  DoS  Joint  Regional 
Strategies,  which  are  geographically  aligned  differently  that  the  GCC  boundaries. 
However,  the  ICS  may  provide  a  pathway  to  a  more  common  lexicon  regarding 
performance  assessment  -  which  could  baseline  interagency  assumptions  and 
expectations,  better  synchronize  activities,  and  converge  goals  at  the  country  level. 

Recommendations 

Based  on  the  methodology  discussion  earlier,  there  is  benefit  in  quantitative 
analysis  and  benefit  in  qualitative  analysis.  Perhaps  the  best  pathway  to  incorporate  the 
best  data  collection  into  security  cooperation  assessment,  then,  is  through  mixed 
methods  data  collecting.  Essential  to  this,  however  is  inclusion  of  assessment  planning 
during  plan  development.  The  planner  should  dedicate  time  to  determining  what 
quantifiable  variables  exist  that  can  be  counted  or  recorded.  Likewise,  the  planner 
should  recognize  where  those  variables  simply  do  not  exist,  and  strive  to  identify 


17 


qualitative  methods  for  monitoring  progress.  Identification  of  these  variables  ahead  of 
time  greatly  simplifies  the  assessment  process  of  “living”  plans  such  as  the  TCP  and 
CCP,  and  meets  the  rapid  update  ability  desired  through  the  APEX  system.  While  it  will 
take  time  to  change  organizational  culture,  implementation  of  operational  design,  and  a 
regular,  ongoing  assessment  of  changes  and  trends  in  the  operational  environment  - 
including  some  qualitative  measures  -  will  lend  itself  well  towards  identifying  these 
variables. 

As  highlighted  early  in  the  paper,  one  key  improvement  to  Security  Cooperation 
assessment  is  better  coordination  and  communication  across  the  interagency.  While 
there  are  provisions  for  this  discussion  at  the  CCDR  level  -  by  means  of  the  PC  -  there 
needs  to  be  improved  synchronization  at  the  individual  country  level.  Perhaps  the  ICS 
construct  will  provide  the  groundwork  for  a  common  lexicon  of  progress  measurement. 

If  some  agreements  can  be  made  regarding  assessment,  then  perhaps  planning  can 
occur  using  a  common  set  of  assumptions  (which  all  agencies  strive  to  validate).  The 
advantage  to  establishing  a  common  construct  would  be  the  improvement  of  integration 
of  country  plans  (currently  the  “maturity  of  integration  is  theater  dependant  and 
inconsistent”36).  Additionally,  if  the  assessment  framework  and  method  is  developed  and 
planned  at  the  ICS  and  CCP  level  and  transferrable  between  the  two,  then  the  workload 
on  already  heavily  tasked  SCOs  would  not  particularly  have  to  increase.  There  is  merit 
for  the  DoD  entities  involved  in  security  cooperation  to  continue  involvement  in  the 
3DPG,  and  become  familiar  with  the  ICS  -  looking  for  parallels  and  areas  for 
cooperative  opportunity. 
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Future  Research  /  Future  Development 
Establishing  the  “common  language”  of  assessment  and  identification  of 
performance  measures  and  indicators  should  not  be  laid  on  the  shoulders  of  those 
already  busy  maintaining  timelines  for  plan  development.  If  the  construct  is  to  be 
common  across  geographic  boundaries  -  and  it  should  since  no  agency  maintains  the 
same  boundaries  -  then  the  coordination  and  development  should  be  kept  at  a  higher 
level.  Perhaps  this  should  be  left  to  the  3DPG;  however,  there  also  exists  room  for 
academic  pursuits  for  identifying  common  variables  based  on  commonality  of  many 
security  cooperation  programs  and  actions.  Future  study  could  compare  programs  and 
actions  in  various  countries  and  identify  overlap  (excess)  or  gaps  in  “known” 
performance  measures.  Additionally,  effort  should  be  given  to  further  define  better 
mixed-method  indicators  for  IMET  programs  to  ensure  dollars  are  being  spent  with  the 
best  effect  in  mind,  not  simply  the  best  efficiency. 

Conclusion 

There  is  need  for  the  development  of  a  standardized  Security  Cooperation 
assessment  framework  compatible  across  both  geographic  boundaries  and  inter  agency 
boundaries.  This  assessment  framework  should  strive  to  include  both  qualitative  and 
quantitative  information  to  provide  the  most  complete  progress  report,  and  should  be 
useful  to  multiple  agencies  in  report  generation  -  unified  effort  preventing  duplication  of 
processes.  Through  these  efforts,  security  cooperation  professionals  will  gain  the  ability 
to  not  only  provide  more  valuable  advice  to  senior  DoD  leaders,  but  also  to  identify 
opportunities  for  cooperation  and  collaboration  among  the  agencies  working  at  the 
country  team  level.  Developing  a  military  culture  of  operational  design  begins  to  build 
the  capacity  to  recognize  and  leverage  qualitative  assessment  processes  and  builds  a 
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more  comprehensive  understanding  of  parallel  lines  of  effort  and  opportunities  for 
cooperation  within  the  inter-agency.  Leveraging  the  new  construct  of  the  Integrated 
Country  Strategy  is  a  starting  point  in  building  better  cooperation.  Providing  the 
overarching  priorities  identified  by  the  Chief  of  Mission,  it  is  the  logical  point  to  build 
from.  As  the  US  military  continues  to  refocus  their  stable-state  operations  more  and 
more  on  security  cooperation  missions,  improvement  in  assessment  methodology 
stands  to  provide  it  with  the  same  efficiency  in  planning  that  it  is  known  for  in  combat 
operations. 
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